Flight path determination

ABSTRACT

A method of determining a flight path for an aerial vehicle, includes controlling the aerial vehicle to fly along a first route, identifying, during the flight along the first route and with aid of one or more processors, a change in a state of signal transmission occurring at a first location, in response to identifying the change, determining, by the one or more processors, a second location different from the first location, determining a second route to the second location, and controlling, by the one or more processors, the aerial vehicle to fly to and land at the second location. The change of the state of signal transmission indicates an abnormal state in a signal transmission between the aerial vehicle and a control device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/406,887, filed on May 8, 2019, which is a continuation ofInternational Application No. PCT/CN2016/105804, filed on Nov. 14, 2016,the entire contents of both of which are incorporated herein byreference.

BACKGROUND

Unmanned vehicles such as unmanned aerial vehicles (UAVs) can be usedfor performing surveillance, reconnaissance, and exploration tasks in awide variety of environments for military and civilian applications. AUAV may be manually controlled by a remote user, or may operate in asemi-autonomous or fully autonomous manner. Such UAVs can includeprocessors to automatically determine how its flight paths.

Existing approaches for determining flight paths with respect tospecific objectives may be less than optimal in some instances. Forexample, the determination may not be sufficiently adaptive to changesof circumstances or tailored to properties of the flying regions.

SUMMARY

This application provides a system and related methods for determiningflight paths within an air space for an aerial vehicle. Thedetermination can be made based on various conditions, such as theconditions of the aerial vehicle, of the flight environment, of signaltransmission between the aerial vehicle and a remote device, and so on.These conditions may generally determine in which regions of the airspace a flight path may lie and in what manner a flight path may bebuilt.

In some aspect, a method of determining a flight path for an aerialvehicle is provided. The method provides identifying during flight andwith aid of one or more processors, a change in a state of signaltransmission occurring at a first location; in response to theidentifying, selecting a first destination within a proximity of asecond location, wherein a state of signal transmission at the secondlocation during a previous flight is different from a state of signaltransmission at the first location; determining a first flight path toreach the first destination, wherein the first flight path comprisesaccessible locations detected by one or more sensors onboard one or moreaerial vehicles; upon reaching the first destination by the aerialvehicle, assessing in real time a state of signal transmission at thefirst destination; and determining a second flight path to a seconddestination based on said assessment.

In some embodiments, the change in a state of signal transmissionincludes a loss or weakening of signal transmitted between the aerialvehicle and a remote device. In some embodiments, the signal transmittedincludes data relating to control signals generated by the remote devicefor operating said aerial vehicle, or image data acquired by the aerialvehicle.

In some embodiments, the first destination is proximal to the secondlocation such that it is within about 10 meters away from the secondlocation. In some embodiments, the first destination is proximal to thesecond location such that signal transmission rates at the firstlocation and the second location are both above a predeterminedthreshold. In some embodiments, the first destination is at the secondlocation. In some embodiments, the change in a state of signaltransmission includes a change in signal transmission rate between theaerial vehicle and a remote device. In some embodiments, the remotedevice is a remote controller or a satellite. In some embodiments, thechange results in the signal transmission rate being below apredetermined threshold.

In some embodiments, the identifying includes receiving a notificationfrom a sensor of the aerial vehicle or a remote device. In someembodiments, the first destination is selected based on a user input. Insome embodiments, the change of a state of signal transmission indicatesan abnormal state in a signal transmission between the aerial vehicleand a remote device. In some embodiments, the first destination isselected based on an operational status of the one or more sensorsonboard the aerial vehicle. In some embodiments, the one or more sensorscomprise one or more GPS receivers. In some embodiments, the firstdestination is selected based on current condition of a flightenvironment. In some embodiments, the current condition of a flightenvironment comprises weather condition or a wireless signal strength.

In some embodiments, the second location is a location detected by theone or more sensors onboard the one or more aerial vehicles during aprevious flight. In some embodiments, the second location is a lastpoint where signal transmission between the aerial vehicle and theremote device was successful. In some embodiments, the first destinationis within a predetermined distance from the first location. In someembodiments, the predetermined distance is less than 10 meters.

In some embodiments, the first flight path is a reverse of a last flightpath of the aerial vehicle. In some embodiments, the first flight pathcomprises no locations undetected by the one or more sensors during theprevious flight. In some embodiments, the first location, firstdestination, second location, or the second destination is characterizedby GPS coordinates. In some embodiments, the one or more sensors onboardthe one or more aerial vehicles comprise one or more of a camera, aradar, a lidar, an ultrasonic sensor, and a GPS receiver. In someembodiments, the second destination is selected based on a user inputfrom a remote device.

In some embodiments, the method further comprises, prior to orconcurrent with selecting the second destination, hovering the aerialvehicle for a predetermined period of time to gather data in real time.In some embodiments, the second flight path is determined while theaerial vehicle is hovering. In some embodiments, when no user input isreceived within the predetermined period of time, the second destinationis a starting point for a current flight. In some embodiments, when nouser input is received within the predetermined period of time,selecting the second destination includes further hovering for a secondpredetermined period of time. In some embodiments, the seconddestination is a location undetected by the one or more sensors duringthe previous flight. In some embodiments, the second destination is apredetermined location for the aerial vehicle to reach beforeidentifying the change in the state of signal transmission occurring atthe first location.

In some embodiments, the method further comprises selecting the seconddestination based on real-time information. In some embodiments, thereal-time information is related to one or more of the following: a userinput, an operational status of the aerial vehicle, and a currentcondition of a flight environment. In some embodiments, the currentcondition of the flight environment includes a weather condition or awireless signal strength. In some embodiments, the real-time informationindicates an abnormal operational status of the aerial vehicle.

In some embodiments, the second destination is a starting point forcurrent flight. In some embodiments, the second destination is a servicepoint providing services to restore a normal operational status, andwherein the second flight path is determined based on a flying distanceto the service point. In some embodiments, the abnormal operationalstatus indicates a low fuel level or a low battery level. In someembodiments, the change in a state of signal transmission includes aloss or weakening of signal transmitted between the aerial vehicle and aremote device. In some embodiments, when the signal transmission remainslost or weakened at the first destination, determining the second flightpath comprises including a location undetected by any of the one or moresensors of the one or more aerial vehicles during previous flight. Insome embodiments, the second flight path includes a location detected byany of the one or more sensors of the one or more aerial vehicles duringa previous flight, when the signal transmission remains lost or weakenedat the first destination. In some embodiments, the second flight pathincludes a location undetected by any of the one or more sensors of theone or more aerial vehicles during a previous flight, when the state ofsignal transmission of the first destination returns to a normal stateat the first destination.

In some embodiments, the method further comprises selecting a secondflight path is further dependent on a user input from a remote device.In some embodiments, wherein the signal transmission includestransmitting image data acquired by an image device onboard the aerialvehicle. In some embodiments, the operational status of the aerialvehicle indicates that power level of a battery configured to power atleast one or more propulsion units of the aerial vehicle falls below athreshold. In some embodiments, the power level is insufficient to powerthe aerial vehicle from the first destination to the second destination.

In another aspect, a system for determining a flight path for an aerialvehicle is provided. The system comprises one or more processors; andone or more memories having instructions stored thereon which whenexecuted by the one or more processors cause the processor to perform:identifying during flight a change in a state of signal transmissionoccurring at a first location; in response to the identifying, selectinga first destination within a proximity of a second location, wherein astate of signal transmission at the second location during a previousflight is different from a state of signal transmission at the firstlocation; determining a first flight path to reach the firstdestination, wherein the first flight path comprises accessiblelocations detected by one or more sensors onboard one or more aerialvehicles; upon reaching the first destination by the aerial vehicle,assessing in real time state of signal transmission at the firstdestination; and determining a second flight path to a seconddestination based on said assessment.

In another aspect, a non-transitory computer-readable storage mediumwith instructions stored thereon that, when executed by a computingsystem, cause the computing system to perform a method of determining aflight path for an aerial vehicle. The method comprises the methodcomprises identifying during flight a change in a state of signaltransmission occurring at a first location; in response to theidentifying, selecting a first destination within a proximity of asecond location, wherein a state of signal transmission at the secondlocation during a previous flight is different from a state of signaltransmission at the first location; determining a first flight path toreach the first destination, wherein the first flight path comprisesaccessible locations detected by one or more sensors onboard one or moreaerial vehicles; upon reaching the first destination by the aerialvehicle, assessing in real time state of signal transmission at thefirst destination; and determining a second flight path to a seconddestination based on said assessment.

In another aspect, a method of planning a flight path for an aerialvehicle is provided. The method comprises obtaining, with aid of one ormore processors, (a) one or more costs each associated with a pathsegment connecting a first point and a second point, wherein the firstpoint and the second point are in a search space of a plurality ofpoints including a starting point and an end point, and (b) one or morecosts each associated with an auxiliary segment connecting the secondpoint and a target point, the target point includes a two-dimensional(2D) coordinate; applying an objective function to the second point,wherein the objective function yields an estimated cost for a route fromthe starting point through the second point to the end point, andwherein the objective function is based on a combination of two or morecomponents, including at least one of the obtained one or more costsassociated with the auxiliary segment; and including the path segmentinto the flight path that starts from the starting point and connects toone or more path segments in a sequence, when the applying of theobjective function to the second point yields a desired value, therebyplanning the flight path.

In some embodiments, the method further comprises repeating the steps ofthe obtaining, the applying and the including until the flight pathincludes the end point. In some embodiments, the desired value is aminimum of values yielded by applying the objective function to multiplecandidate second points in the search space. In some embodiments, thetwo or more components further include one or more costs associated witha straight line connecting the second point and the end point.

In some embodiments, the method further comprises directing the aerialvehicle to follow the flight path. In some embodiments, the 2Dcoordinate for the target point is associated with a surface height withrespect to a reference level for which an actual value is unknown to theaerial vehicle. In some embodiments, the target point is a nearest pointto the second point.

In some embodiments, the cost associated with the auxiliary segment isrelated to a category of an object associated with the target point. Insome embodiments, the cost associated with the auxiliary segment isrelated to a distance between the second point and the target point. Insome embodiments, the cost associated with the auxiliary segment isrelated to an estimate of a surface height at the target point or aconfidence indicator associated with the estimate. In some embodiments,the method further comprises assigning a weight to the cost associatedwith the auxiliary segment depending on the confidence indicatorassociated with the estimate in computing a value of the objectivefunction.

In some embodiments, the cost associated with the path segment from thefirst point to the second point is related to a distance between the twopoints. In some embodiments, the cost associated with the path segmentis further related to a difference between the heights of the two pointswhen the height of the second point is greater than the height of thefirst point. In some embodiments, the method further comprises assigninga first weight to a distance between 2D coordinates corresponding to thetwo points and a second weight to a difference in the third dimension ofthe two points in computing a value of the cost associated with the pathsegment.

In some embodiments, the cost associated with the path segment isrelated to one or more of the following status indicators associatedwith the aerial vehicle: an aggregate battery level, an aggregate GPSsignal strength, an aggregate ground control signal strength, and anaggregate image transmission rate over points on the path segment.

In some embodiments, the method further comprises determining a costassociated with a path segment connecting a preceding point of the firstpoint and the second point that are along a line of sight. In someembodiments, the path segment connecting a preceding point of the firstpoint and the second point is added to the flight path when the costassociated with the path segment connecting a preceding point of thefirst point and the second point is less than an aggregated costassociated with the path segment connecting the first point and thesecond point and a path segment connecting the preceding point of thefirst point and the first point. In some embodiments, the determiningthe cost associated with the path segment connecting a preceding pointof the first point and the second point is performed only when the costof the auxiliary segment from the second point to the target point isgreater than a predetermined threshold. In some embodiments, the cost ofthe path segment connecting a preceding point of the first point and thesecond point is related to a distance from the preceding point of thefirst point to the second point. In some embodiments, the method furthercomprises determining if an obstacle is present in the path segmentconnecting a preceding point of the first point and the second point.

In some embodiments, the method further comprises, upon detecting apresence of a predetermined condition in real time, directing the aerialvehicle to follow a new flight path. In some embodiments, thepredetermined condition is emergence of an unforeseen obstacle or lossof a signal.

In some embodiments, the end point is a previously visited point, aknown safe point, a ground control center, or a user location.

In another aspect, a system for planning a flight path for an aerialvehicle is provided. The system comprises one or more processors; andone or more memories having instructions stored thereon which whenexecuted by the processor cause the processor to perform: obtaining (a)one or more costs each associated with a path segment connecting a firstpoint and a second point, wherein the first point and the second pointare in a space of a plurality of points including a starting point andan end point, and (b) one or more costs each associated with anauxiliary segment connecting the second point and a target point, thetarget point includes a two-dimensional (2D) coordinate; applying anobjective function to the second point, wherein the objective functionyields an estimated cost for a route from the starting point through thesecond point to the end point, and wherein the objective function isbased on a combination of two or more components, including at least oneof the obtained one or more costs associated with the auxiliary segment;and including the path segment into the flight path that starts from thestarting point and connects to one or more path segments in a sequence,when the applying of the objective function to the second point yields adesired value, thereby planning the flight path.

In another aspect, a non-transitory computer-readable storage mediumwith instructions stored thereon that, when executed by a computingsystem, cause the computing system to perform a method of planning aflight path for an aerial vehicle. The method comprises obtaining (a)one or more costs each associated with a path segment connecting a firstpoint and a second point, wherein the first point and the second pointare in a space of a plurality of points including a starting point andan end point, and (b) one or more costs each associated with anauxiliary segment connecting the second point and a target point, thetarget point includes a two-dimensional (2D) coordinate; applying anobjective function to the second point, wherein the objective functionyields an estimated cost for a route from the starting point through thesecond point to the end point, and wherein the objective function isbased on a combination of two or more components, including at least oneof the obtained one or more costs associated with the auxiliary segment;and including the path segment into the flight path that starts from thestarting point and connects to one or more path segments in a sequence,when the applying of the objective function to the second point yields adesired value, thereby planning the flight path.

In yet another aspect, a method of planning a flight path for an aerialvehicle is provided. The method comprises identifying, with aid of oneor more processors, a plurality of candidate points, wherein theplurality of candidate points are in a predetermined relationship with afirst point; determining a cost associated with each candidate point ofthe plurality of candidate points, wherein the determining of the costis at least based on a distance between each candidate point and atarget point, wherein the target point is associated with a surfaceheight with respect to a reference level; selecting one candidate pointfrom the plurality candidate points, when the cost associated with theselected candidate point satisfies a predetermined condition; andincluding a segment connecting the first point and the selectedcandidate point into the flight path, thereby planning the flight path.

In some embodiments, the method further comprises taking the selectedcandidate point as the first point and repeating the steps of theidentifying, the determining, the selecting and the including until theflight path reaches an end point. In some embodiments, the methodfurther comprises directing the aerial vehicle to follow the flightpath.

In some embodiments, the plurality of candidate points are within apredetermined distance relative to the first point. In some embodiments,the predetermined condition indicates a minimum of costs associated withthe plurality of candidate points. In some embodiments, the determiningof the cost is further based on a distance between each candidate pointand the first point. In some embodiments, the determining of the costassociated with one of the candidate points is further based on adifference between the height of the one candidate point and the firstpoint, when the height of the one candidate point is greater than theheight of the first point.

In some embodiments, the determining of the cost is further based on adistance between each candidate point and an end point. In someembodiments, the determining of the cost is further based on one or morestatus indicators associated with the aerial vehicle during a previousflight at the each candidate point. In some embodiments, a height of asurface point corresponding to the target point is unknown to the aerialvehicle. In some embodiments, the target point for a particularcandidate point is a nearest point to the particular candidate point ofwhich a surface height is unknown to the aerial vehicle.

In some embodiments, the method further comprises determining a firstcost associated with a segment connecting a preceding point of the firstpoint and the selected candidate point; determining a second costassociated with a segment connecting the preceding point of the firstpoint and the first point; determining a third cost associated with asegment connecting the first point and the selected candidate point; andincluding a segment connecting the preceding point of the first pointand selected candidate point into the flight path, when the first costis lower than a summation of the second cost and the third cost. In someembodiments, the method further comprises determining if the segmentconnecting the preceding point of the first point and the selectedcandidate point is obstacle free.

In another aspect, a system for planning a flight path for an aerialvehicle is provided. The system comprises one or more processors; andone or more memories having instructions stored thereon which whenexecuted by the one or more processors cause the processor to perform:identifying a plurality of candidate points, wherein the plurality ofcandidate points are in a predetermined relationship with a first point;determining a cost associated with each candidate point of the pluralityof candidate points, wherein the determining of the cost is at leastbased on a distance between each candidate point and a target point,wherein the target point is associated with a surface height withrespect to a reference level; selecting one candidate point from theplurality candidate points, when the cost associated with the selectedcandidate point satisfies a predetermined condition; and including asegment connecting the first point and the selected candidate point intothe flight path, thereby planning the flight path.

In another aspect, a non-transitory computer-readable storage mediumwith instructions stored thereon that, when executed by a computingsystem, cause the computing system to perform a method of planning aflight path for an aerial vehicle. The method comprises identifying aplurality of candidate points, wherein the plurality of candidate pointsare in a predetermined relationship with a first point; determining acost associated with each candidate point of the plurality of candidatepoints, wherein the determining of the cost is at least based on adistance between each candidate point and a target point, wherein thetarget point is associated with a surface height with respect to areference level; selecting one candidate point from the pluralitycandidate points, when the cost associated with the selected candidatepoint satisfies a predetermined condition; and including a segmentconnecting the first point and the selected candidate point into theflight path, thereby planning the flight path.

It shall be understood that different aspects of the disclosure can beappreciated individually, collectively, or in combination with eachother. Various aspects of the disclosure described herein may be appliedto any of the particular applications set forth below or for any othertypes of movable objects. Any description herein of an aerial vehiclemay apply to and be used for any movable object, such as any vehicle.Additionally, the systems, devices, and methods disclosed herein in thecontext of aerial motion (e.g., flight) may also be applied in thecontext of other types of motion, such as movement on the ground or onwater, underwater motion, or motion in space. Furthermore, anydescription herein of a rotor or rotor assembly may apply to and be usedfor any propulsion system, device, or mechanism configured to generate apropulsive force by rotation (e.g., propellers, wheels, axles).

Other objects and features of the present disclosure will becomeapparent by a review of the specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present disclosure will be obtained by reference tothe following detailed description that sets forth illustrativeembodiments, in which the principles of the disclosure are utilized, andthe accompanying drawings of which:

FIG. 1 illustrates a UAV operating in an outdoor environment, inaccordance with embodiments;

FIG. 2 illustrates a UAV operating in an indoor environment, inaccordance with embodiments;

FIG. 3 illustrates how an UAV obtains data for a 2D elevation map.

FIG. 4 illustrates a 2D elevation map based on data collected by afront-view sensor.

FIG. 5 illustrates a scenario where multiple surface heights exist for a2D coordinate.

FIG. 6 illustrates an example process performed by a UAV in response toan abnormal condition.

FIG. 7 illustrates an example process of determining a flight path in aregion with no obstacle.

FIG. 8 illustrates an example process of determining a flight path in aregion with an obstacle.

FIG. 9 illustrates another example process of determining a flight pathin a region with an obstacle.

FIG. 10 illustrates an example approach of using different path searchalgorithms for different regions.

FIG. 11 illustrates an example process performed by a UAV to determine aflight path based on costs derived from a 2D elevation map.

FIG. 12 illustrates an example process performed by a UAV to determine aflight path by determining connected sub flight paths through differentregions using different path search algorithm.

FIG. 13 illustrates a UAV, in accordance with embodiments;

FIG. 14 illustrates a movable object including a carrier and a payload,in accordance with embodiments; and

FIG. 15 illustrates a system for controlling a movable object, inaccordance with embodiments.

DETAILED DESCRIPTION

Systems and methods are provided for controlling movable objects such asan unmanned aerial vehicle (UAV). In some embodiments, the UAV can beadapted to carry a plurality of sensors configured to collectenvironmental data. Some of the sensors may be of different types (e.g.,a vision sensor used in combination with a proximity sensor). The dataobtained by the plurality of sensors can be combined to generate anenvironmental map representative of the surrounding environment. In someembodiments, the environmental map can include information regarding thelocation of objects including obstacles in the environment. The UAV canuse the generated map to perform various operations, some of which maybe semi-automated or fully automated. For instance, the environmentalmap can be used to automatically determine a flight path for the UAV tonavigate from its current location to a target location. Theenvironmental map may be useful when the UAV performs an auto-returnfunction from its current location to a start position, or any otherspecified position. As another example, the environmental map can beused to determine the spatial disposition of one or more obstacles andthereby enable the UAV to perform obstacle avoidance maneuvers.Advantageously, the use of multiple sensor types for collectingenvironmental data as disclosed herein can improve the accuracy ofenvironmental mapping even in diverse environments and operatingconditions, thereby enhancing the robustness and flexibility of UAVfunctionalities such as navigation and obstacle avoidance.

The embodiments provided herein can be applied to various types of UAVs.For instance, the UAV may be a small-scale UAV that weighs no more than10 kg and/or has a maximum dimension of no more than 1.5 m. In someembodiments, the UAV may be a rotorcraft, such as a multi-rotor aircraftthat is propelled to move through the air by a plurality of propellers(e.g., a quadcopter). Additional examples of UAVs and other movableobjects suitable for use with the embodiments presented herein aredescribed in further detail below.

The UAVs described herein can be operated completely autonomously (e.g.,by a suitable computing system such as an onboard controller),semi-autonomously, or manually (e.g., by a human user). The UAV canreceive commands from a suitable entity (e.g., human user or autonomouscontrol system) and respond to such commands by performing one or moreactions. For example, the UAV can be controlled to take off from theground, move within the air (e.g., with up to three degrees of freedomin translation and up to three degrees of freedom in rotation), move totarget location or to a sequence of target locations, hover within theair, land on the ground, and so on. As another example, the UAV can becontrolled to move at a specified velocity and/or acceleration (e.g.,with up to three degrees of freedom in translation and up to threedegrees of freedom in rotation) or along a specified movement path.Furthermore, the commands can be used to control one or more UAVcomponents, such as the components described herein (e.g., sensors,actuators, propulsion units, payload, etc.). For instance, some commandscan be used to control the position, orientation, and/or operation of aUAV payload such as a camera. Optionally, the UAV can be configured tooperate in accordance with one or more predetermined operating rules.The operating rules may be used to control any suitable aspect of theUAV, such as the position (e.g., latitude, longitude, altitude),orientation (e.g., roll, pitch yaw), velocity (e.g., translationaland/or angular), and/or acceleration (e.g., translational and/orangular) of the UAV. For instance, the operating rules can be designedsuch that the UAV is not permitted to fly beyond a threshold height,e.g., the UAV can be configured to fly at a height of no more than 400 mfrom the ground. In some embodiments, the operating rules can be adaptedto provide automated mechanisms for improving UAV safety and preventingsafety incidents. For example, the UAV can be configured to detect arestricted flight region (e.g., an airport) and not fly within apredetermined distance of the restricted flight region, thereby avertingpotential collisions with aircraft and other obstacles.

Turning now the drawings, FIG. 1 illustrates a UAV 102 operating in anoutdoor environment 100, in accordance with embodiments. The outdoorenvironment 100 may be an urban, suburban, or rural setting, or anyother environment that is not at least partially within a building. TheUAV 102 may be operated relatively close to the ground 104 (e.g., lowaltitude) or relatively far from the ground 104 (e.g., high altitude).For example, a UAV 102 operating less than or equal to approximately 10m from the ground may be considered to be at low altitude, while a UAV102 operating at greater than or equal to approximately 10 m from theground may be considered to be at high altitude.

In some embodiments, the outdoor environment 100 includes one or moreobstacles 108 a-d. An obstacle may include any object or entity that mayobstruct the movement of the UAV 102. Some obstacles may be situated onthe ground 104 (e.g., obstacles 108 a, 108 d), such as buildings, groundvehicles (e.g., cars, motorcycles, trucks, bicycles), human beings,animals, plants (e.g., trees, bushes), and other manmade or naturalstructures. Some obstacles may be in contact with and/or supported bythe ground 104, water, manmade structures, or natural structures.Alternatively, some obstacles may be wholly located in the air 106(e.g., obstacles 108 b, 108 c), including aerial vehicles (e.g.,airplanes, helicopters, hot air balloons, other UAVs) or birds. Aerialobstacles may not be supported by the ground 104, or by water, or by anynatural or manmade structures. An obstacle located on the ground 104 mayinclude portions that extend substantially into the air 106 (e.g., tallstructures such as towers, skyscrapers, lamp posts, radio towers, powerlines, trees, etc.).

FIG. 2 illustrates a UAV 152 operating in an indoor environment 250, inaccordance with embodiments. The indoor environment 250 is within theinterior of a building 254 having a floor 256, one or more walls 258,and/or a ceiling or roof 260. Exemplary buildings include residential,commercial, or industrial buildings such as houses, apartments, offices,manufacturing facilities, storage facilities, and so on. The interior ofthe building 254 may be completely enclosed by the floor 256, walls 258,and ceiling 260 such that the UAV 252 is constrained to the interiorspace. Conversely, at least one of the floor 256, walls 258, or ceiling260 may be absent, thereby enabling the UAV 252 to fly from inside tooutside, or vice-versa. Alternatively or in combination, one or moreapertures 264 may be formed in the floor 256, walls 258, or ceiling 260(e.g., a door, window, skylight).

Similar to the outdoor environment 100, the indoor environment 250 caninclude one or more obstacles 262 a-d. Some obstacles may be situated onthe floor 256 (e.g., obstacle 262 a), such as furniture, appliances,human beings, animals, plants, and other manmade or natural objects.Conversely, some obstacles may be located in the air (e.g., obstacle 262b), such as birds or other UAVs. Some obstacles in the indoorenvironment 250 can be supported by other structures or objects.Obstacles may also be attached to the ceiling 260 (e.g., obstacle 262c), such as light fixtures, ceiling fans, beams, or otherceiling-mounted appliances or structures. In some embodiments, obstaclesmay be attached to the walls 258 (e.g., obstacle 262 d), such as lightfixtures, shelves, cabinets, and other wall-mounted appliances orstructures. Notably, the structural components of the building 254 canalso be considered to be obstacles, including the floor 256, walls 258,and ceiling 260.

The obstacles described herein may be substantially stationary (e.g.,buildings, plants, structures) or substantially mobile (e.g., humanbeings, animals, vehicles, or other objects capable of movement). Someobstacles may include a combination of stationary and mobile components(e.g., a windmill). Mobile obstacles or obstacle components may moveaccording to a predetermined or predictable path or pattern. Forexample, the movement of a car may be relatively predictable (e.g.,according to the shape of the road). Alternatively, some mobileobstacles or obstacle components may move along random or otherwiseunpredictable trajectories. For example, a living being such as ananimal may move in a relatively unpredictable manner.

In order to ensure safe and efficient operation, it may be beneficial toprovide the UAV with mechanisms for detecting and identifyingenvironmental objects such as obstacles. Additionally, recognition ofenvironmental objects such as landmarks and features can facilitatenavigation, particularly when the UAV is operating in a semi-autonomousor fully autonomous manner. Furthermore, knowledge of the UAV's preciselocation within the environment, as well as its spatial relationship tosurrounding environmental objects, can be valuable for a wide variety ofUAV functionalities.

Accordingly, the UAVs described herein can include one or more sensorsconfigured to collect relevant data, such as information relating to theUAV state, the surrounding environment, or the objects within theenvironment. Exemplary sensors suitable for use with the embodimentsdisclosed herein include location sensors (e.g., global positioningsystem (GPS) sensors, mobile device transmitters enabling locationtriangulation), vision sensors (e.g., imaging devices capable ofdetecting visible, infrared, or ultraviolet light, such as cameras),proximity or range sensors (e.g., ultrasonic sensors, lidar,time-of-flight or depth cameras), inertial sensors (e.g.,accelerometers, gyroscopes, inertial measurement units (IMUs)), altitudesensors, attitude sensors (e.g., compasses) pressure sensors (e.g.,barometers), audio sensors (e.g., microphones) or field sensors (e.g.,magnetometers, electromagnetic sensors). Any suitable number andcombination of sensors can be used, such as one, two, three, four, five,or more sensors. Optionally, the data can be received from sensors ofdifferent types (e.g., two, three, four, five, or more types). Sensorsof different types may measure different types of signals or information(e.g., position, orientation, velocity, acceleration, proximity,pressure, etc.) and/or utilize different types of measurement techniquesto obtain data. For instance, the sensors may include any suitablecombination of active sensors (e.g., sensors that generate and measureenergy from their own energy source) and passive sensors (e.g., sensorsthat detect available energy). As another example, some sensors maygenerate absolute measurement data that is provided in terms of a globalcoordinate system (e.g., position data provided by a GPS sensor,attitude data provided by a compass or magnetometer), while othersensors may generate relative measurement data that is provided in termsof a local coordinate system (e.g., relative angular velocity providedby a gyroscope; relative translational acceleration provided by anaccelerometer; relative attitude information provided by a visionsensor; relative distance information provided by an ultrasonic sensor,lidar, or time-of-flight camera). In some instances, the localcoordinate system may be a body coordinate system that is definedrelative to the UAV.

The sensors described herein can be carried by the UAV. A sensor can besituated on any suitable portion of the UAV, such as above, underneath,on the side(s) of, or within a vehicle body of the UAV. Some sensors canbe mechanically coupled to the UAV such that the spatial dispositionand/or motion of the UAV correspond to the spatial disposition and/ormotion of the sensors. The sensor can be coupled to the UAV via a rigidcoupling, such that the sensor does not move relative to the portion ofthe UAV to which it is attached. Alternatively, the coupling between thesensor and the UAV can permit movement of the sensor relative to theUAV. The coupling can be a permanent coupling or non-permanent (e.g.,releasable) coupling. Suitable coupling methods can include adhesives,bonding, welding, and/or fasteners (e.g., screws, nails, pins, etc.).Optionally, the sensor can be integrally formed with a portion of theUAV. Furthermore, the sensor can be electrically coupled with a portionof the UAV (e.g., processing unit, control system, data storage) so asto enable the data collected by the sensor to be used for variousfunctions of the UAV (e.g., navigation, control, propulsion,communication with a user or other device, etc.), such as theembodiments discussed herein.

The sensors can be configured to collect various types of data, such asdata relating to the UAV, the surrounding environment, or objects withinthe environment. For example, at least some of the sensors may beconfigured to provide data regarding a state of the UAV. The stateinformation provided by a sensor can include information regarding aspatial disposition of the UAV (e.g., location or position informationsuch as longitude, latitude, and/or altitude; orientation or attitudeinformation such as roll, pitch, and/or yaw). The state information canalso include information regarding motion of the UAV (e.g.,translational velocity, translation acceleration, angular velocity,angular acceleration, etc.). A sensor can be configured, for instance,to determine a spatial disposition and/or motion of the UAV with respectto up to six degrees of freedom (e.g., three degrees of freedom inposition and/or translation, three degrees of freedom in orientationand/or rotation). The state information may be provided relative to aglobal coordinate system or relative to a local coordinate system (e.g.,relative to the UAV or another entity). For example, a sensor can beconfigured to determine the distance between the UAV and the usercontrolling the UAV, or the distance between the UAV and the startingpoint of flight for the UAV.

The data obtained by the sensors may provide various types ofenvironmental information. For example, the sensor data may beindicative of an environment type, such as an indoor environment,outdoor environment, low altitude environment, or high altitudeenvironment. The sensor data may also provide information regardingcurrent environmental conditions, including weather (e.g., clear, rainy,snowing), visibility conditions, wind speed, time of day, and so on.Furthermore, the environmental information collected by the sensors mayinclude information regarding the objects in the environment, such asthe obstacles described herein. Obstacle information may includeinformation regarding the number, density, geometry, and/or spatialdisposition of obstacles in the environment.

In some embodiments, sensing results are generated by combining sensordata obtained by multiple sensors, also known as “sensor fusion.” Forinstance, sensor fusion can be used to combine sensing data obtained bydifferent sensor types, including as GPS sensors, inertial sensors,vision sensors, lidar, ultrasonic sensors, and so on. As anotherexample, sensor fusion can be used to combine different types of sensingdata, such as absolute measurement data (e.g., data provided relative toa global coordinate system such as GPS data) and relative measurementdata (e.g., data provided relative to a local coordinate system such asvision sensing data, lidar data, or ultrasonic sensing data). Sensorfusion can be used to compensate for limitations or inaccuraciesassociated with individual sensor types, thereby improving the accuracyand reliability of the final sensing result.

In some embodiments, a UAV can maintain communication with a controldevice, typically operated by a user on the ground. Indirect or directcommunications between the UAV and the control device may be provided.For example, the UAV may maintain indirect communication with thecontrol device over a communication network, such as a cellular network.In another example, the UAV may maintain direct communication with thecontrol device via a direct, short-range wireless communication channel.

One-way or two-way communications may be provided between the UAV andthe control device. Typically, the control device transmits controlsignals to the UAV to guide the flight of the UAV. The control data fromthe control device may control operation of the UAV, and/or a payloadcarried by the UAV. Operation of the UAV propulsion units, flightcontroller, navigation module, one or more sensors, a carrier of theUAV, landing stand of the UAV, and/or communication module of the UAVmay be controlled.

The UAV may transmit data generated by its sensors, motor, or otherinformation to the control device, which may be useful in furtherguiding the flight. Any description herein of sensor data may includeany data from the UAV to the control device including, but not limitedto, data collected by sensors about an external environment, and/oroperational status data about the UAV. Data about the externalenvironment of the UAV may include, but is not limited to image data,distance or depth profiles, satellite signals, and so on. Operationalstatus data about the UAV may include, but is not limited to,information about power provided to propulsion units of the UAV, powerconsumed by the propulsion units, navigation of the UAV, attitude of theUAV, disposition of the carrier, disposition of a payload, communicationof the UAV, or energy storage and/or consumption of the UAV. As anexample, operational status data indicates an internal condition of theUAV, such as that the energy level is adequate or the battery level islow. As another example, sensor data indicates a current condition ofthe flight environment, which may include an obstacle ahead that doesnot exist on an existing map, as discussed above, or an inclementweather condition. The transmission is typically performed periodicallybut can be adjusted as necessary.

In some embodiments, a UAV is configured to collect data during itsflight and builds 2D or 3D maps based on the collected data. As oneexample, the UAV can track various types of wireless transmission. TheUAV may track wireless signals in order to monitor wirelesscommunication environment. For example, the UAV may track incoming GPSsignals, incoming control signals, outgoing data signals, etc. The UAVmay track signals that are used for navigation, flight control, sensors,image transmission, or any other type of communication. The UAV maytrack whether a wireless communication channel is available, bandwidth,signal strength, signal to noise ratio, and/or any other aspect ofwireless transmission. The UAV may track wireless transmission atdifferent locations. A resulting signal map can provide the UAV withreference signal strengths in a particular air space. As anotherexample, the UAV can keep track of the presence or absence of any objectin the air space. In addition to data collected by its sensors, the UAVcan incorporate data collected by other UAVs or recorded in other datasources before or during the flight or at any time. For example, variousterrain maps and weather maps are available for public consumption. TheUAV can also update its maps for specific coordinates based on new datagenerated by its sensors or obtained from other sources at any time.

In some embodiments, in order for a UAV to have a higher flyingefficiency, the UAV can include only one sensor for a reduced weight ordesign complexity. The sensor can be located in the front with adetection range in front of the UAV. For example, a camera can capture aview with a proper resolution, or a laser can detect distance to objectswith a sufficient accuracy, within the detection range that can becharacterized by a certain height, width, and depth with respect to theposition of the sensor. In addition, the UAV can include a single sensorthat is positioned on the back or the side of the UAV where it can havea proper detection range. The sensor data can be processed to build a 2Delevation map, which indicates height information for each point in a 2Dcoordinate system with respect to a reference level, such as the groundlevel.

FIG. 3 illustrates how an UAV obtains data for a 2D elevation map. Anobject 320 that extends from a reference level 330 at the point 350intersects the detection range 340. However, only a mid-portion of theobject 320 is in the detection range, while the top of the object 320 isnot. By analyzing the data generated by the sensor 315 corresponding tothe detection range 340, the UAV can conclude that the top of the object320 is higher than the top of the detection range 360, or in otherwords, the distance from top of the detection range 360 to the referencelevel is a minimum value for the surface height at the point 350.Therefore, the UAV can store in the elevation map that distance as anestimate of the surface height at the point 350 having a 2D coordinatewith respect to the reference level and a category indicating that theestimated value is a lower threshold or a minimum value for the surfaceheight. The UAV can similarly create data for the 2D elevation map whenthe top of an object is within or below the detection range. Detailsregarding the construction and maintenance of 2D elevation maps can befound in the co-pending application titled “Construction and Update ofElevation Maps” filed on Nov. 14, 2016, which is hereby incorporated byreference in its entirety. Alternatively, a plurality of sensors may beprovided on the UAV. The different sensors may have different fields ofview.

In some embodiments, the resulting 2D elevation map includes informationin addition to an estimated height for each 2D coordinate, such as aconfidence indicator, including a risk of crashing into an obstacleregarding the estimate. For example, the 2D elevation map can indicate,for each 2D coordinate, a height estimated from certain sensor data,together with a category indicating whether the estimated height wouldbe the actual value (“green category” hereinafter), a minimum value(“red category” hereinafter), or a maximum value (“blue category”hereinafter) of an actual surface height at the 2D coordinate. The 2Delevation map can also indicate, for a coordinate with no informationavailable, a default or a maximum value as the estimated height and acategory indicating that the estimated height is of no use (“blankcategory” hereinafter). The 2D elevation map can also include confidenceindicators or other indicators of the quality of the estimate, such ascapabilities of sensor, visibility at the time sensor data wasgenerated, etc. The confidence indicators can be implemented by numbers,strings, or other ordinal values. In some embodiments, the confidenceindicators can be represented as the categories. Different values forthe confidence indicators can represent different levels of risk ofobstruction or danger. For example, the blank, red, blue, and greencategories indicate increasing levels of risk of obstruction or danger.In another example where the confidence indicators are represented bynumeric values, a higher confidence indicator can represent a higherrisk, whereas a lower confidence indicator can represent a lower risk.Alternatively, a lower confidence indicator can represent a higher risk,and vice versa. Such a 2D elevation map is relatively easy to build andmaintain and can be used to avoid obstacles or other undesirablelocations in determining flight paths.

FIG. 4 illustrates a 2D elevation map based on data collected by afront-view sensor. The landscape shows the estimated heights for the 2Dcoordinates on the ground, while the coloring of the landscape shows theconfidence indicators for the estimated surface heights of thecorresponding 2D coordinates. The coloring can also indicate thecategories for the estimated surface heights. For example, the blackcolor can correspond to the red category, the dark gray color cancorrespond to the green category, the light gray color can correspond tothe blue category, and no or blank color can correspond to the blankcategory. Therefore, the 2D coordinate for the point 402 has a redcategory, meaning that the surface height for the point 402 is theminimum of an actual surface height at the 2D coordinate. Similarly, the2D coordinate for the dark gray point 404 has a green category, the 2Dcoordinate for the light gray point 406 has a blue category, and the 2Dcoordinate for the white point 408 has a blank category. Such a visualrepresentation can be useful to a user controlling the UAV.

In some embodiments, the 2D elevation map includes, in addition to anestimated height for each 2D coordinate and a confidence indicatorassociated with the estimated surface height, a height range to whichthe estimated surface height belongs. The height range information canbe especially useful for 2D coordinates having multiple surface heightsdue to existence of objects that do not extend all the way from theground.

FIG. 5 illustrates a scenario where multiple surface heights exist for a2D coordinate. The estimated surface height and the confidence scoredetermined for the point 502 under the bridge 504 could be differentdepending on a height range. For example, a UAV might determine that thesurface height for the point 502 within a height range between 0 m and10 m does not exceed 10 m, while the a UAV might determine that thesurface for the point 502 within a height range between 10 m and 20 m isat least 20 m. Therefore, given a 2D elevation map that includes aheight range associated with an estimated height for a 2D coordinate, aUAV can compare the height range with its current flying altitude tobetter determine its flight paths, possibly taking advantage ofadditional flying space, such as the area under a bridge. For example,the height range can be selected based on a current altitude of the UAV.For example, a first height range of 0 m to 10 m may be selected if thealtitude of the UAV falls between 0 m and 10 m above ground; a secondheight range of 10 m to 20 m may be selected if the altitude of the UAVfalls between 10 m and 20 m above ground. Depending on the selectedheight range, different estimated surface heights and confidenceindicators may be determined for the same 2D point. A flight path withrespect to a given 2D point can be determined based on the estimatedsurface height and the confidence indicator for the 2D point (2Delevation data), taking into consideration the height range associatedwith the 2D elevation data and/or an altitude of the UAV.

In some embodiments, a UAV can classify each point in a region based onwhat types of information is available for the location. For example,the UAV may distinguish those points that have been detected by itssensors during the current flight, during a previous flight (which maybe any flight made before the current point in time, any flight beforethe UAV made the last stop, etc.), or during any flight by one of agroup of predetermined UAVs that belong to the same organization or haveaccess to the same elevation maps, or a combination thereof. The UAV mayalso distinguish those points for which any information has beencollected in any manner from those points for which no information hasever been collected. The availability of information makes thedistinguished points “safe” in a way, although the degree of safety canvary depending on who generated the information, how the information wasgenerated, when the information was generated, and so on. Generally, apoint is not safe to a UAV unless it is accessible or reachable to theUAV. A point may be unreachable to the UAV due to the nature of thepoint (e.g., presence of obstacles or bad weather), the operationalstate of the UAV (e.g., lack of fuel), etc. In addition, certain pointscan be designated for special purposes. For example, the starting pointof the current flight or a list of usual takeoff and landing points canbe determined and marked in a map. A list of service points where theUAV can be serviced—receiving energy or other supplies, replacementparts, check-ups, tune-ups, repairs, and so on can be identified.

In some embodiments, an airborne UAV may need to find a new flight pathin response to an abnormal and often emergency situation. For example,data transmission from or to the UAV may be intermittent or otherwiseinterrupted, resulting in weak or disappearing signals. When a user iscontrolling flight of the UAV from a remote control, loss of the controlsignal may mean that the UAV needs to auto-pilot or risk ending in acrash. When navigation signals, such as GPS are lost, the UAV may nothave an accurate indication of the UAV's location. Alternatively, theUAV may have to rely on other navigation techniques to ascertain theUAV's location which may be less accurate (e.g., may “drift” over time)or may use more energy. In some instances, data transmission by the UAV,such as image transmission, may be lost, which may present morechallenges for a user who is trying to control the UAV remotely. Weathercondition may be stormy or otherwise affecting vehicle operation,including sensor performance. The batteries on the UAV may becomedepleted during flight, and the UAV may be reaching a low energy levelbeyond which the UAV may not have enough energy to return. Vehicle partsmay be failing due to mechanical, electrical, or other problems, such aspart breakage detachment, circuit errors, battery drains, etc.

In any of these scenarios, it would be highly desirable to first restorethe normal condition, often by finding a new flight path to a possiblynew destination. For example, upon determining that wirelesstransmission is intermittent or non-existent, the UAV would want toreach an area where wireless transmission is successful anduninterrupted. The determination can be made from a failure to receivenormal control signals from a control device or GPS signals from thesatellites, a failure to transmit relevant data to the control device,repeated notifications from the control device for receiving no orlittle data, etc. for a specific period of time. Similarly, the UAVwould want to reach an area where the air condition is calm and clear orwhere vehicle services are available.

In some embodiments, when the communication channel between a UAV and acontrol device is not broken, especially when transmission of controlsignals continues to be acceptable, the control device can instruct theUAV to follow a specific flight path to reach a specific destination,such as the starting point (flight take-off point) or one of thedesignated points where the abnormal condition is likely to be resolved.A designated point can be the last known location of the user (e.g.,based on location of the remote controller), a pre-set “home” pointdesignated by the user, an earlier location on the current flight orprevious flight, etc. When the communication channel is broken, however,the UAV could automatically determine a flight path to reach anappropriate destination. The UAV may be able to autonomously orsemi-autonomously initiate and/or execute the process to reach theappropriate destination. The UAV may initiate and/or execute the processto reach the appropriate destination without human aid or intervention,and/or any instructions from any devices external to the UAV.

As previously described, a UAV may initiate a process to reach theappropriate destination in response to one or more detected conditions.The one or more detected conditions may fall into one or more differentreason categories. In some embodiments, a plurality of reason categoriesmay be predetermined. Examples of reason categories may include, but arenot limited to, user instruction, GPS signal loss, remote controllersignal loss, image transmission loss, other wireless signal loss, lowbattery, and/or component failure. The UAV may identify which reasoncategory that a detected condition falls under. Depending on the reasoncategory, the UAV may select a flight mode to reach the appropriatedestination. Further descriptions of flight modes are provided ingreater detail elsewhere herein. In some embodiments, a single flightmode may correspond to a single reason category. Alternatively, multipleflight modes may correspond to a single reason category, or singleflight modes may correspond to multiple reason categories.

In some embodiments, a UAV is configured to automatically determine aflight path in response to an abnormal condition based on a variety offactors without human intervention, including an automatic return to oneof the locations detected by onboard sensors since take-off. The UAV canfly in any of different navigation modes. For example, the UAV can fly a“safe” mode where the flight path is limited to “safe” points for whichadequate information for safe flight is available based on sensor datapreviously generated by this or other UAVs. Such information can berelated to the presence or absence of obstacles, climate conditions,crowdedness, etc. The UAV can also fly in an “exploring” mode where theflight path is not so limited and can be flexible to achieve specificobjectives, such as flying the shortest distance. It is possible to havea layered structure for different navigation modes, where starting withthe safe mode, each mode is less restrictive than the next, based on thenumber of points in space that can be considered in determining a flightpath, the amount of time it takes to reach the next destination, and soon. It is also possible to have different navigation modes for achievingdifferent objectives.

In some embodiments, to restore the normal condition as soon or easilyas possible, the UAV can choose to limit the upcoming flight to the“safe” points for which adequate information is available, as discussedabove, thereby flying in a safe mode. Specifically, the UAV may searchamong the safe points in finding a desirable flight path, so that theUAV will follow a flight path along these safe points. This approachtends to reduce the search space as well as the likelihood ofaggravating the abnormal condition. In the safe mode, one simple flightpath is a reverse route. For example, after failing to transmit data toa control device for a period of time, the UAV can backtrack to a pointwhere wireless transmission strength was strong, a region where datatransmission rate was above a threshold, or any other safe pointaccording to the maps. The flight path can be systematically determinedin other ways, as discussed below. The UAV can dynamically determinewhen to enter or leave a mode based on real-time information related touser input, environment conditions, UAV operational status, etc.

In some embodiments, once the abnormal condition is resolved, the UAV isconfigured to determine another flight path in real time. At this point,it is less important to limit the flight to safe points, and thus theUAV can be flexible in determining its flight path, thereby entering anexploring mode. The UAV can hover to gather and assess data, or it canincrementally determine the flight path as it travels. The hovering cancontinue until the receipt of user instructions from a control device,until the determination of a new flight path, until the end of aspecific period of time, etc. The user instructions may determine thesubsequent flight path. Otherwise, the UAV may select the nextdestination and determine a corresponding flight path based on thecurrent condition of the environment or the vehicle. As one example,after a struggle through a bad-signal region, the energy or batterylevel may be low, and the UAV can then select a service stop for arefill or recharge. As another example, it might be important to reachthe original destination to be reached before the abnormal conditionarose, and the UAV can return to the flight path it was following beforethe abnormal condition arose. As yet another example, there might belittle time left to go anywhere, and then the UAV can head straight backto the starting point.

In some embodiments, upon reaching a safe point that was selected forrestoring the normal condition, the abnormal condition may persist. TheUAV can then repeatedly select additional safe points as subsequentdestinations until the abnormal condition is resolved or some othercriterion is met, such as when the number of repeated attempts or theamount of time spending on repeated attempts exceeds a predeterminedthreshold. The number of repeated attempts to make can be determinedbased on the distances to know safe points, the location of the UAV, thetime remaining before the UAV is required to return to the take-offpoint, etc. Alternatively, as the safe points no longer prove to beadvantageous, the UAV can enter the exploring mode until a specificcriterion is met or stay in the exploring mode until it reaches thefinal destination.

FIG. 6 illustrates an example process performed by a UAV in response toan abnormal condition. In step 602, the UAV identifies an abnormal stateof signal transmission (at a first location). The identification can bebased on data generated by its sensors or motor. The abnormal statecould be a loss or weakening of signals, which could be measured by thesignal transmission rate over a period of time, possibly with respect toa minimum threshold. In step 604, the UAV selects a first destinationsuch that signal transmission was deemed normal at the first destinationor at a second location that is proximal to the first destination by thesensors of this or other UAVs at an earlier time. The proximity can bedetermined with respect to a predetermined distance. In step 606, theUAV determines a first path to the first destination. In someembodiments, the UAV travels to the first destination in the safe mode,which means that the UAV searches among only the “safe” points, thoselocations previously detected by the sensors of this or other UAVs, inbuilding the first flight path. In step 608, upon following the firstflight path to reach the first destination, the UAV assesses the stateof signal transmission at the first destination.

In steps 610 and 612, the UAV determines the second destination in realtime based on the current state of signal transmission at the firstdestination and determines the second flight path to reach the seconddestination. The determination can also be based on other currentconditions, such as the control state of the remote device, the weather,the operational status of the aerial vehicle, etc. If the current stateof signal transmission is normal as expected, the UAV can be lessrestrictive in selecting the second destination and the second flightpath. Specifically, the UAV may no longer stay in the safe mode. In theabsence of any commands from a user, the UAV can determine the best nextdestination. For example, if the current battery level is low, the UAVcan select a service stop or a home base as the second destination anddetermines the quickest way to reach the second destination as thesecond flight path. On the other hand, if the current state of signaltransmission is still not satisfactory, the UAV can repeatedly selectanother safe point as the second destination until the current signalcondition is back to normal. The UAV can stay in the safe mode duringthis process. Alternatively, the UAV can immediately enter the exploringmode and choose the second destination based on real-time information.

In some embodiments, the UAV can generally follow a planned flight path.However, the UAV can also build a flight path in real time by repeatedlyselecting a nearby intermediate destination or a mere flying directioneach time without pre-selecting all the points on the flight path atonce. The extent of “real time”, which is generally deemed to depictimmediate feedback without any purposeful delay, can be measured withinminutes, seconds, tenths of seconds, etc. By building a flight path inthis manner, the UAV can be highly adaptive and may better respond toabnormal conditions. For example, onboard sensors can generally detectobjects that would constitute obstacles that need to be avoided for theUAV. Stationary obstacles may already be known, but moving obstacles mayemerge. In addition, routine motor inspection may have been performed,but undetected or unexpected motor errors may arise. Therefore, upondetecting adverse situations, such as a motor failure or an obstacle inthe way, the UAV can break from the planned flight path and build a newflight path in real time. When the adverse situations are overcome, theUAV can then return to the planned flight path as appropriate. On theother hand, the continuous need to determine next steps may cause lesscomputing resources to be available for other tasks during the flight.It may thus be desirable for the UAV to select a few more destinationsat a time or make less frequent real-time decisions, especially when thedemand for computing resources is high.

In some embodiments, as long as control signals from a control devicecontinue to be received by the UAV, the control device can continue todirect the UAV to follow specific flight paths. In cases where the UAVhas automatically determined a flight path, the UAV can choose tooverride the control signals until it reaches the end of the flight pathor some intermediate destination. This approach may be advantageous whenthe UAV needs to act quickly and avoid any delays, which may occurduring the communication with the control device. Alternatively, the UAVcan abandon the automatically determined flight path upon receiving acontrol signal so as to follow the instructions issued by the controldevice.

In some embodiments, a UAV may need to automatically determine a flightpath through an air space. The UAV may determine a new flight path uponentering a new navigation mode, entering a distinct geographical region,encountering an abnormal condition, receiving instructions to reach anew destination, etc. The UAV may focus on predetermined points withinthe air space or regards the air space as a grid of points. The UAV canthen choose a flight path through one or more of these points withrespect to specific objectives. Example objectives include reducing thetotal flight time or distance, avoiding regions more likely to containobstacles, and avoiding changes in flying altitude. To enable systematicsearches for a desirable flight path, the UAV can build a cost functionc(s, s′) by assigning a numerical cost to a segment between any two ofthese points s ands' depending on the specific objectives. Such asegment can represent a path segment that a UAV can follow in movingbetween the points s and s′. For example, the costs can be related tothe length of the segment (distance between endpoints of the segment),the average energy consumption rate over the segment, the landscapealong the segment, or other characteristics of the segment. The UAV canalso assign a cost to each of these points. For example, the costs canbe related to the likely presence of an obstacle, the typicaltemperature at the point, and the distance from the point to any “safe”point, etc. In general, any factor that may affect the “quality” offlight can be represented in a cost function, such as the wirelesssignal (control, GPS, data, etc.) transmission state (e.g., transmissionrate, signal strengths, signal-to-noise ratio, etc.), weather condition(rain, snow, fog, hurricane, etc.), climate condition (wind, pressure,temperature, brightness, etc.), gravitational effect, and so on. Asearch can then be performed typically by building a route with areduced or (local or global) minimum cost aggregated over the costs ofthe segments and points in the growing route. Generally, the costfunction is built and the costs are assigned (obtained) in advance, andthe costs can be looked up (obtained) during the search for a desirableflight path. For example, the cost function can be built for one regionat a time, to be used for subsequent searches within that region. It ispossible, however, to dynamically calculate (obtain) the cost for asegment every time the segment is considered in the search.

In some embodiments, a UAV is configured to assign costs based on a 2Delevation map discussed above. The cost can depend on the estimatedheight as well as the risk of encountering obstacles or one or moreconfidence indicators for each 2D coordinate in various ways. Forexample, a 2D coordinate having a larger estimated height, a largerheight range known for the estimated height, a lower confidenceindicator associated with the estimate, or any combination thereof maybe considered less favorable for a flight path and hence lead to ahigher cost. A point having such a 2D coordinate (sometimes called a“target point” below) should generally be avoided or assigned a lowerpriority. For example, any point in space that can be mapped to a 2Dcoordinate in the elevation map that is associated with a confidenceindicator indicating a red category, such as a point on the object 402in FIG. 4 (especially those below the estimated surface height), can beconsidered as part of an obstacle to be avoided in the flight path. As aresult, a larger cost may be assigned a point or a segment which whenprojected to the reference level is closer to more of these unfavorable2D coordinates. The cost can further depend on a category of an objectassociated with the target point. For example, a higher cost may beassigned to a point when its corresponding target

In some embodiments, a UAV can incorporate costs determined from a 2Delevation map into the cost function in different ways. It's generallypreferable to stay away from those points that are more likely tocontain obstacles. Therefore, the UAV can associate more costs withbeing in proximity to those points where the estimated heights are abovea certain threshold or where the categories are red or blank for thecorresponding 2D coordinates in the 2D elevation map. In a moreelaborate scheme, larger costs can be attached to areas close to thosepoints where the estimated heights are higher the current flyingaltitude and those points where the estimated heights are no higher thanthe currently flying altitude and the categories are red or blank. Otherschemes are possible. The proximity can be related to the physicaldistance, the travelling time, etc. Varying the cost by proximity tospecific 2D coordinates means that such cost tends to peak around thespecific 2D coordinates and gradually decrease as other coordinates thatare less and less proximate to the specific 2D coordinates. It enablesthe selection of flight paths that have less proximity to those 2Dcoordinates.

To incorporate costs for proximity to 2D coordinates in the redcategory, for example, the cost assigned to each segment from a firstpoint to a second point can be adjusted to be related to a combination(such as a sum) of the physical distance between the two points and thephysical distance from the segment to the 2D coordinates in the redcategory. Generally, a search space can include a set of points within aregion including a departure point and an arrival point for an UAV. Therelationship between a point in the search space and a 2D coordinate inthe red category or any point of interest that is not physically in thesearch space can be represented as an auxiliary segment, for example,and the cost attached to being proximal to a 2D coordinate in the redcategory may then be associated with such an auxiliary segment. Such anauxiliary segment is used to represent a relationship with a point ofinterest, such as proximity, with which a cost can be associated; it canbut does not need to correspond to a path segment. A separate auxiliarysegment can be used to represent proximity to each point of interest, ora single auxiliary segment can be used to represent overall proximity toall points of interest. For example, the overall proximity cancorrespond to the distance to the closest point of interest, or theaverage distance over the set of points associated with the red categoryand within a certain radius from the particular point. The distancebetween a segment and a specific point can be computed in various ways,such as a shortest distance or an average distance between the specificpoint and any point on the segment. Mathematically, incorporating costsfor proximity to 2D coordinates can be done by updating an existing costfunction c(s, s′) or adding another cost function red_cost(s, s′).Alternatively, the cost assigned to each point (one of the endpoints ofa segment) can be adjusted to be related to the physical distance fromthe point to 2D coordinates in the red category. Additional costs couldbe incorporated for proximity to 2D coordinates in the red and blankcategories, a weighted proximity to 2D coordinates in differentcategories, or in additional ways. For instance, the weights coulddecrease for the blank, red, blue, and green categories, respectively.Additional costs could also be incorporated for proximity to otherpoints generally to be avoided, such as points on human bodies to avoidcollisions with humans or points that have not been detected by anyknown sensors onboard UAVs.

In some embodiments, a UAV can maintain a constant altitude to reducegravitational effects and possible complications from changingaltitudes. Specifically, searching for flight paths through pointshaving different heights can require more computational resources andresult in more complex flight paths than searching through points havingthe same or similar altitude. To avoid changes in flying altitude, upondetecting an obstacle, the UAV can decide to fly around it whilemaintaining the flying altitude rather than over it by increasing theflying altitude. However, it may be desirable to increase the flyingaltitude in some cases, such as when the width of the obstacle at thecurrent flying altitude is much larger than the height of the obstaclewith respect to the current flying altitude.

In some embodiments, a UAV is configured to incorporate costs based onheight information for the points in the search space in different ways.For example, a higher cost may be associated with flying to a higher ordifferent altitude. Similar to incorporating costs for proximity to 2Dcoordinates in certain categories on a 2D elevation map, the costassigned to each segment or point can be adjusted to cover the costrelated to the flying altitude. For instance, the original cost assignedto a segment from a first point to a second point can be the physicaldistance between the corresponding 2D coordinates at the ground level,and an additional cost can be added during execution of the path searchalgorithm when the increase in height from the first point to the secondpoint is greater than a certain threshold. In addition, the UAV canweigh the vertical component and the horizontal components of the costdifferently. For example, the UAV can assign a smaller weight to thecost associated with the distance between the corresponding 2Dcoordinates but a larger weight to the cost associated with thedifference in heights.

In some embodiments, the UAV can utilize the flying altitudes includedin 2D elevation maps in determining flight paths. For example, referringback to FIG. 5 , the 2D elevation maps may include, in the form of(height range, category), (0 m˜10 m, green), (10 m˜20 m, red), and (20m˜30 m, blue). The 2D coordinate with (10 m˜20 m, red), such as 502,would no longer correspond to an obstacle to the UAV when the UAV isflying at 8 m. Similarly, when the current flying altitude is 15 m, (10m˜20 m, red) would matter more to the UAV than (0 m˜10 m, green) and (20m˜30 m, blue). One approach is to give priority to elevation data havingheight ranges that are closer to the current or desired flying altitude,such as by selecting the category associated with the height range thatis closest to the current or desired flying altitude out of all theelevation data for a 2D coordinate. For example, when the UAV is flyingat 8 m, green would be selected as the category for the point 502 andthus all points within 0 m˜10 m above 502, and thus flying near any ofthese points would not be penalized. Clearly, this is a safe approachonly when the UAV is to maintain more or less the same flying altitude.Another approach is to incorporate all elevation data. For example,regardless of the current flying altitude, the UAV treats the pointsabove the 502 as being associated with different categories and couldassign costs accordingly. The UAV may then have choices for its flyingaltitude, although in practice changing its flying altitude to 20 mwould be discouraged. Yet another approach is to dynamically prioritizeelevation data during execution of a path search algorithm based on theheight of the last point on a growing route for the next point to beadded to the growing route.

In some embodiments, a UAV can employ any path search algorithms knownto someone of ordinary skill in the art, including A*, Theta*, D*, andtheir variants, to systematically determine a superior flight path froma source (starting point) to a destination (end point) using anobjective function that represents the cost of a path from the startingpoint to the end point. Such a path search algorithm can operate byiteratively minimizing the value of the objective function overdifferent routes from the starting point in the search space, with thevalue of the objective function increasingly closer to the actual costof an actual path from the starting point to the end point. For example,the A* algorithm works by minimizing the following objective functionƒ(n)=g(n)+h(n), where n is the last point on a growing route, g(n) isthe cost of a route from the start point to n, and h(n) is a heuristicthat estimates the cost of the cheapest route from n to the end point.g(n) and h(n) can be combined by summation or other aggregation. Thecost of a route can be the sum or other aggregation of costs assigned toall the segments and/or all the points on the route. The A* algorithmbuilds one or more growing routes until one of them reaches the endpoint, and the growing route that reaches the end point has the minimumpossible cost when h(n) is admissible. Specifically, g(n) can be brokendown into g(n's parent)+c(n's parent, n), where the parent refers to theprevious or preceding point on the growing route; h(n) is often simplythe (straight-line) distance from n to the end point. At each iteration,an n is identified and added to a growing route. Specifically, a set ofcandidates for n is identified. These candidates are typically neighborsof the last points on the existing growing routes (so connected to thoselast points by segments), but additional criteria can be imposed onchoosing the candidates. For example, the set of candidates can belimited to those where the costs of the corresponding segments (whichcan correspond to the length of the segment or distance between the twopoints of the segment at both ends) are below a predetermined threshold.The candidate that leads to a minimal value of the objective functionamong all the candidates is then added to the growing route.

The Theta* algorithm works in a similar fashion except that it considersthe presence of a line of sight and may replace existing segments withline-of-sight segments in the growing route. In one version, the Theta*algorithm considers path segments between the parent of the parent of nand n (referred as line-of-sight segments) at each iteration andreplaces the two segments from the parent of the parent to n by acorresponding line-of-sight segment when the cost of the two segments isgreater than the cost of the line-of-sight segment. Typically, the lineof sight is considered for every candidate, and then the candidate thatleads to a minimal value of the objective function among all thecandidates is then added to the growing route as in A*. It is possible,however, to first identify n from the set of candidates as in A*, andconsider whether to incorporate the line-of-sight segment for only theidentified n instead.

In some embodiments, a UAV can determine flight paths using differentapproaches in different regions. For example, the UAV can switch betweenA* and Theta*. The main difference between the A* and Theta* is that aflight path is built from predetermined segments between points in A*,while a flight path can include line-of-sight segments in Theta*. Sincethe line-of-sight segments essentially skip points, Theta* normally runsfaster than A*. FIG. 7 illustrates an example process of determining aflight path in a region with no obstacle. A* could be used to find ashortest flight path from the source 702 to the destination 708, shownas the dotted line in FIG. 7 . FIG. 8 illustrates an example process ofdetermining a flight path in a region with an obstacle. The obstacle canbe around points 804 and 806, and there may no longer be a segment fromthe point 804 to the point 806. The UAV can then assign additional coststo points or segments depending on how close they are to the points 804and 806. A* can then be used to find a flight path that is short andaway from the obstacle, shown as the dotted line from the point 802through the points 810 and 812 to the point 808 in FIG. 8 . FIG. 9illustrates another example process of determining a flight path in aregion with an obstacle, shown as the dotted line from the point 902through the point 910 to the point 908. Theta* can be used instead tofind a flight path that is short and away from the obstacle. Instead ofapplying A* or Theta* to find a route in a region, it is also possibleto consider line-of-sight segments dynamically during the path-searchingprocess based on how close the current point is to an obstacle. Forexample, while applying the A* algorithm, upon determining that n is notclose to an obstacle, the system can temporarily consider the presenceof a line of sight between the parent of the parent of n and n, or viceversa.

In some embodiments, a UAV is configured to choose which path searchalgorithm to use depending on whether the region to be searched includeor are near known obstacles. As can be seen, while the flight path foundby Theta* may be shorter than the flight path found by A*, it can becloser to the obstacle, especially when additional costs are associatedwith points rather than segments. Since the original routes aregenerally shorter than the line-of-sight routes, it may be easier tohave finer control of the selection of the flight path. In addition, itmay be easier to add costs to the points in the search space, which areconsidered regardless of which path search algorithm is used, than tothe additional line-of-sight segments which are only used in Theta*.Therefore, the UAV can choose to utilize A* in regions around knownobstacles and Theta* in other regions. FIG. 10 illustrates an exampleapproach of using different path search algorithms for differentregions. The box represents an air space, and the coloring (black, darkgray, light gray, and white) indicates, for the points in the air space,the categories of the corresponding 2D coordinates in the 2D elevationmap. To go from the source 1028 to the destination 1030 through the airspace, the UAV can first identify every region 1022 that may be within acertain distance away from where obstacles are likely to be present,such as those points 1026 where the corresponding 2D coordinates areassociated with the red category in the 2D elevation map. The UAV canthen find a flight path using A* to in that region 1022 and using Theta*in other regions 1020 and 1024. In this case, for each region, the UAVcan select a set of possible regional sources and a set of possibleregional destinations such that any flight path from the regional sourceto the region al destination runs in the general direction towards the(final) destination. The UAV can then take a simply flight path betweenthe last regional destination and the current regional source, forexample. In a simpler scheme, the UAV can first select a set ofintermediate points 1032 and 1034, such as a set of points detected bysensors during previous flight. The UAV can then generally break the airspace into regions 1020, 1022, and 1024 along these intermediate points.Depending on the likelihood that obstacles are present in each region,which may be indicated by the number of points where the corresponding2D coordinates are associated with the red category, the UAV can thenchoose an appropriate path search algorithm. More generally, dependingon the properties of different path search algorithms and the propertiesof different regions within the given air space, the UAV can chooseother combinations of algorithms for different regions, differentnavigation modes, different reasons for changing flight paths, etc.

FIG. 11 illustrates an example process performed by a UAV to determine aflight path based on costs derived from a 2D elevation map. The UAVstarts with a set of points and a set of directed segments connectingsome of these points in an air space. A directed segment normally existsbetween two points when they are considered adjacent to each other. Thesegment is typically directed as the cost of flying from point A topoint B may be different from that of flying from point B to point A.The UAV then builds a cost function, which could incorporate differenttypes of information, such as information available in a 2D elevationmap discussed above. Specific costs can be assigned to the directedsegments between adjacent points, to the individual points, toadditional line-of-sight segments, and so on. In steps 1102 and 1104,the UAV builds the cost function. In step 1102, the UAV assigns a costto each existing segment between each ordered pair of adjacent points(each directed segment) in the air space. The assignment could be basedany combination of relevant factors, including the physical distance,the average battery level, the average GPS signal strength, the averagesignal transmission rate between the UAV and a remote device, etc. Instep 1104, the UAV further assigns a cost to each point based on itsrelationship, such as proximity, to specific “dangerous” points, such asthose whose 2D coordinates are associated with the red category in the2D elevation map. The relationship between a point and a dangerous pointcan be represented by an auxiliary segment between the two points.Alternatively, the cost of each segment can be calculated every time thesegment is to be considered in the search for a flight path. In step1106, the UAV then executes a path search algorithm, such as A*, Theta*,or any variant, using the cost function it has built. To execute Theta*,the UAV needs to take into consideration additional line-of-sightsegments and the costs associated with these segments. The UAV canpre-assign these additional costs to the line-of-sight segments orcompute them on the fly, during the execution of the Theta* algorithm.

FIG. 12 illustrates an example process performed by a UAV to determine aflight path by determining connected sub flight paths through differentregions using different path search algorithm. Essentially, Step 1106 inFIG. 11 can be performed by this process. In steps 1202 and 1204, theUAV identifies intermediate points or intermediate destinations and usesthese intermediate points to break the search space into regions. Eachof the regions would then be at least partially defined by twointermediate points, or one intermediate point and the source or thedestination. In an alternative approach, the UAV identifies the regionsfirst and uses these regions to identify intermediate points. In step1206, the UAV classifies each of the regions based on one or factors,such as the likelihood that obstacles are present in the region. Forexample, the UAV can classify a region that contains a point whose 2Dcoordinate is associated with a red category as a “dangerous” region,and any other region as a “safe” region. The UAV would also associateeach class with a specific path search algorithm. For example, adangerous region can be associated with A*, while a safe region can beassociated with Theta*. In step 1208, the UAV determines a flight pathfor each region between its two defining points using the path searchalgorithm associated with the class of the region to determine theflight path for the UAV through the air space.

FIG. 13 illustrates an unmanned aerial vehicle (UAV) 800, in accordancewith embodiments of the present disclosure. The UAV may be an example ofa movable object as described herein. The UAV 800 can include apropulsion system having four rotors 1302, 1304, 1306, and 1308. Anynumber of rotors may be provided (e.g., one, two, three, four, five,six, or more). The rotors can be embodiments of the self-tighteningrotors described elsewhere herein. The rotors, rotor assemblies, orother propulsion systems of the unmanned aerial vehicle may enable theunmanned aerial vehicle to hover/maintain position, change orientation,and/or change location. The distance between shafts of opposite rotorscan be any suitable length 1310. For example, the length 1310 can beless than or equal to 2 m, or less than equal to 5 m. In someembodiments, the length 1310 can be within a range from 40 cm to 1 m,from 10 cm to 2 m, or from 5 cm to 5 m. Any description herein of a UAVmay apply to a movable object, such as a movable object of a differenttype, and vice versa.

In some embodiments, the movable object can be configured to carry aload. The load can include one or more of passengers, cargo, equipment,instruments, and the like. The load can be provided within a housing.The housing may be separate from a housing of the movable object, or bepart of a housing for a movable object. Alternatively, the load can beprovided with a housing while the movable object does not have ahousing. Alternatively, portions of the load or the entire load can beprovided without a housing. The load can be rigidly fixed relative tothe movable object. Optionally, the load can be movable relative to themovable object (e.g., translatable or rotatable relative to the movableobject).

In some embodiments, the load includes a payload. The payload can beconfigured not to perform any operation or function. Alternatively, thepayload can be a payload configured to perform an operation or function,also known as a functional payload. For example, the payload can includeone or more sensors for surveying one or more targets. Any suitablesensor can be incorporated into the payload, such as an image capturedevice (e.g., a camera), an audio capture device (e.g., a parabolicmicrophone), an infrared imaging device, or an ultraviolet imagingdevice. The sensor can provide static sensing data (e.g., a photograph)or dynamic sensing data (e.g., a video). In some embodiments, the sensorprovides sensing data for the target of the payload. Alternatively or incombination, the payload can include one or more emitters for providingsignals to one or more targets. Any suitable emitter can be used, suchas an illumination source or a sound source. In some embodiments, thepayload includes one or more transceivers, such as for communicationwith a module remote from the movable object. Optionally, the payloadcan be configured to interact with the environment or a target. Forexample, the payload can include a tool, instrument, or mechanismcapable of manipulating objects, such as a robotic arm.

Optionally, the load may include a carrier. The carrier can be providedfor the payload and the payload can be coupled to the movable object viathe carrier, either directly (e.g., directly contacting the movableobject) or indirectly (e.g., not contacting the movable object).Conversely, the payload can be mounted on the movable object withoutrequiring a carrier. The payload can be integrally formed with thecarrier. Alternatively, the payload can be releasably coupled to thecarrier. In some embodiments, the payload can include one or morepayload elements, and one or more of the payload elements can be movablerelative to the movable object and/or the carrier, as described above.

The carrier can be integrally formed with the movable object.Alternatively, the carrier can be releasably coupled to the movableobject. The carrier can be coupled to the movable object directly orindirectly. The carrier can provide support to the payload (e.g., carryat least part of the weight of the payload). The carrier can include asuitable mounting structure (e.g., a gimbal platform) capable ofstabilizing and/or directing the movement of the payload. In someembodiments, the carrier can be adapted to control the state of thepayload (e.g., position and/or orientation) relative to the movableobject. For example, the carrier can be configured to move relative tothe movable object (e.g., with respect to one, two, or three degrees oftranslation and/or one, two, or three degrees of rotation) such that thepayload maintains its position and/or orientation relative to a suitablereference frame regardless of the movement of the movable object. Thereference frame can be a fixed reference frame (e.g., the surroundingenvironment). Alternatively, the reference frame can be a movingreference frame (e.g., the movable object, a payload target).

In some embodiments, the carrier can be configured to permit movement ofthe payload relative to the carrier and/or movable object. The movementcan be a translation with respect to up to three degrees of freedom(e.g., along one, two, or three axes) or a rotation with respect to upto three degrees of freedom (e.g., about one, two, or three axes), orany suitable combination thereof.

In some instances, the carrier can include a carrier frame assembly anda carrier actuation assembly. The carrier frame assembly can providestructural support to the payload. The carrier frame assembly caninclude individual carrier frame components, some of which can bemovable relative to one another. The carrier actuation assembly caninclude one or more actuators (e.g., motors) that actuate movement ofthe individual carrier frame components. The actuators can permit themovement of multiple carrier frame components simultaneously, or may beconfigured to permit the movement of a single carrier frame component ata time. The movement of the carrier frame components can produce acorresponding movement of the payload. For example, the carrieractuation assembly can actuate a rotation of one or more carrier framecomponents about one or more axes of rotation (e.g., roll axis, pitchaxis, or yaw axis). The rotation of the one or more carrier framecomponents can cause a payload to rotate about one or more axes ofrotation relative to the movable object. Alternatively or incombination, the carrier actuation assembly can actuate a translation ofone or more carrier frame components along one or more axes oftranslation, and thereby produce a translation of the payload along oneor more corresponding axes relative to the movable object.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, can be controlled by a terminal. Theterminal can be a remote control device at a location distant from themovable object, carrier, and/or payload. The terminal can be disposed onor affixed to a support platform. Alternatively, the terminal can be ahandheld or wearable device. For example, the terminal can include asmartphone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal can include auser interface, such as a keyboard, mouse, joystick, touchscreen, ordisplay. Any suitable user input can be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal can be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal can be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal can be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal can include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload.

The terminal can include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal can be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal can display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the positioning of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 14 illustrates a movable object 900 including a carrier 1402 and apayload 1404, in accordance with embodiments. Although the movableobject 1400 is depicted as an aircraft, this depiction is not intendedto be limiting, and any suitable type of movable object can be used, aspreviously described herein. One of skill in the art would appreciatethat any of the embodiments described herein in the context of aircraftsystems can be applied to any suitable movable object (e.g., a UAV). Insome instances, the payload 1404 may be provided on the movable object1400 without requiring the carrier 1402. The movable object 1400 mayinclude propulsion mechanisms 1406, a sensing system 1408, and acommunication system 1410.

The propulsion mechanisms 1406 can include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. For example, the propulsion mechanisms 1406 maybe self-tightening rotors, rotor assemblies, or other rotary propulsionunits, as disclosed elsewhere herein. The movable object may have one ormore, two or more, three or more, or four or more propulsion mechanisms.The propulsion mechanisms may all be of the same type. Alternatively,one or more propulsion mechanisms can be different types of propulsionmechanisms. The propulsion mechanisms 1406 can be mounted on the movableobject 1400 using any suitable means, such as a support element (e.g., adrive shaft) as described elsewhere herein. The propulsion mechanisms1406 can be mounted on any suitable portion of the movable object 1400,such on the top, bottom, front, back, sides, or suitable combinationsthereof.

In some embodiments, the propulsion mechanisms 1406 can enable themovable object 1400 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1400 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1406 can be operable to permit the movableobject 1400 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanisms 1400 may becontrolled independently of the other propulsion mechanisms.Alternatively, the propulsion mechanisms 1400 can be configured to becontrolled simultaneously. For example, the movable object 1400 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1400. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally rotors may spin in acounterclockwise direction. For example, the number of clockwise rotorsmay be equal to the number of counterclockwise rotors. The rotation rateof each of the horizontally oriented rotors can be varied independentlyin order to control the lift and/or thrust produced by each rotor, andthereby adjust the spatial disposition, velocity, and/or acceleration ofthe movable object 1400 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation).

The sensing system 1408 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1400 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors caninclude global positioning system (GPS) sensors, motion sensors,inertial sensors, proximity sensors, or image sensors. The sensing dataprovided by the sensing system 1408 can be used to control the spatialdisposition, velocity, and/or orientation of the movable object 1400(e.g., using a suitable processing unit and/or control module, asdescribed below). Alternatively, the sensing system 1408 can be used toprovide data regarding the environment surrounding the movable object,such as weather conditions, proximity to potential obstacles, locationof geographical features, location of manmade structures, and the like.

The communication system 1410 enables communication with terminal 1412having a communication system 1414 via wireless signals 1416. Thecommunication systems 1410, 1414 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication, such that data can betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1400 transmitting data to theterminal 1412, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1410 to one or morereceivers of the communication system 1412, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata can be transmitted in both directions between the movable object1400 and the terminal 1412. The two-way communication can involvetransmitting data from one or more transmitters of the communicationsystem 1410 to one or more receivers of the communication system 1414,and vice-versa.

In some embodiments, the terminal 1412 can provide control data to oneor more of the movable object 1400, carrier 1402, and payload 1404 andreceive information from one or more of the movable object 1400, carrier1402, and payload 1404 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier and/orpayload. For example, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the propulsion mechanisms 1406), or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 1402).The control data from the terminal may result in control of the payload,such as control of the operation of a camera or other image capturingdevice (e.g., taking still or moving pictures, zooming in or out,turning on or off, switching imaging modes, change image resolution,changing focus, changing depth of field, changing exposure time,changing viewing angle or field of view). In some instances, thecommunications from the movable object, carrier and/or payload mayinclude information from one or more sensors (e.g., of the sensingsystem 1408 or of the payload 1404). The communications may includesensed information from one or more different types of sensors (e.g.,GPS sensors, motion sensors, inertial sensor, proximity sensors, orimage sensors). Such information may pertain to the position (e.g.,location, orientation), movement, or acceleration of the movable object,carrier and/or payload. Such information from a payload may include datacaptured by the payload or a sensed state of the payload. The controldata provided transmitted by the terminal 1412 can be configured tocontrol a state of one or more of the movable object 1400, carrier 1402,or payload 1404. Alternatively or in combination, the carrier 1402 andpayload 1404 can also each include a communication module configured tocommunicate with terminal 1412, such that the terminal can communicatewith and control each of the movable object 1400, carrier 1402, andpayload 1404 independently.

In some embodiments, the movable object 1400 can be configured tocommunicate with another remote device in addition to the terminal 1412,or instead of the terminal 1412. The terminal 1412 may also beconfigured to communicate with another remote device as well as themovable object 1400. For example, the movable object 1400 and/orterminal 1412 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicecan be configured to transmit data to the movable object 1400, receivedata from the movable object 1400, transmit data to the terminal 1412,and/or receive data from the terminal 1412. Optionally, the remotedevice can be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1400 and/orterminal 1412 can be uploaded to a website or server.

The systems, devices, and methods described herein can be applied to awide variety of movable objects. As previously mentioned, anydescription herein of a UAV may apply to and be used for any movableobject. A movable object of the present disclosure can be configured tomove within any suitable environment, such as in air (e.g., a fixed-wingaircraft, a rotary-wing aircraft, or an aircraft having neither fixedwings nor rotary wings), in water (e.g., a ship or a submarine), onground (e.g., a motor vehicle, such as a car, truck, bus, van,motorcycle; a movable structure or frame such as a stick, fishing pole;or a train), under the ground (e.g., a subway), in space (e.g., aspaceplane, a satellite, or a probe), or any combination of theseenvironments. The movable object can be a vehicle, such as a vehicledescribed elsewhere herein. The movable object may be a self-propelledunmanned vehicle that does not require human input. In some embodiments,the movable object can be mounted on a living subject, such as a humanor an animal. Suitable animals can include avians, canines, felines,equines, bovines, ovines, porcines, delphines, rodents, or insects. Insome embodiments, the movable object may be carried.

The movable object may be capable of moving freely within theenvironment with respect to six degrees of freedom (e.g., three degreesof freedom in translation and three degrees of freedom in rotation).Alternatively, the movement of the movable object can be constrainedwith respect to one or more degrees of freedom, such as by apredetermined path, track, or orientation. The movement can be actuatedby any suitable actuation mechanism, such as an engine or a motor. Theactuation mechanism of the movable object can be powered by any suitableenergy source, such as electrical energy, magnetic energy, solar energy,wind energy, gravitational energy, chemical energy, nuclear energy, orany suitable combination thereof. The movable object may beself-propelled via a propulsion system, as described elsewhere herein.The propulsion system may optionally run on an energy source, such aselectrical energy, magnetic energy, solar energy, wind energy,gravitational energy, chemical energy, nuclear energy, or any suitablecombination thereof. Alternatively, the movable object may be carried bya living being.

In some instances, the movable object can be a vehicle. Suitablevehicles may include water vehicles, aerial vehicles, space vehicles, orground vehicles. For example, aerial vehicles may be fixed-wing aircraft(e.g., airplane, gliders), rotary-wing aircraft (e.g., helicopters,rotorcraft), aircraft having both fixed wings and rotary wings, oraircraft having neither (e.g., blimps, hot air balloons). A vehicle canbe self-propelled, such as self-propelled through the air, on or inwater, in space, or on or under the ground. A self-propelled vehicle canutilize a propulsion system, such as a propulsion system including oneor more engines, motors, wheels, axles, magnets, rotors, propellers,blades, nozzles, or any suitable combination thereof. In some instances,the propulsion system can be used to enable the movable object to takeoff from a surface, land on a surface, maintain its current positionand/or orientation (e.g., hover), change orientation, and/or changeposition.

The movable object can be controlled remotely by a user or controlledlocally by an occupant within or on the movable object. In someembodiments, the movable object is an unmanned movable object, such as aUAV. An unmanned movable object, such as a UAV, may not have an occupantonboard the movable object. The movable object can be controlled by ahuman or an autonomous control system (e.g., a computer control system),or any suitable combination thereof. The movable object can be anautonomous or semi-autonomous robot, such as a robot configured with anartificial intelligence.

FIG. 15 is a schematic illustration by way of block diagram of a system1500 for controlling a movable object, in accordance with embodiments.The system 1500 can be used in combination with any suitable embodimentof the systems, devices, and methods disclosed herein. The system 1500can include a sensing module 1502, processing unit 1504, non-transitorycomputer readable medium 1506, control module 1508, and communicationmodule 1510.

The sensing module 1502 can utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors can includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1502 can beoperatively coupled to a processing unit 1504 having a plurality ofprocessors. In some embodiments, the sensing module can be operativelycoupled to a transmission module 1512 (e.g., a Wi-Fi image transmissionmodule) configured to directly transmit sensing data to a suitableexternal device or system. For example, the transmission module 1512 canbe used to transmit images captured by a camera of the sensing module1502 to a remote terminal.

The processing unit 1504 can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Theprocessing unit 1504 can be operatively coupled to a non-transitorycomputer readable medium 1506. The non-transitory computer readablemedium 1506 can store logic, code, and/or program instructionsexecutable by the processing unit 1504 for performing one or more steps.The non-transitory computer readable medium can include one or morememory units (e.g., removable media or external storage such as an SDcard or random access memory (RAM)). In some embodiments, data from thesensing module 1502 can be directly conveyed to and stored within thememory units of the non-transitory computer readable medium 1506. Thememory units of the non-transitory computer readable medium 1506 canstore logic, code and/or program instructions executable by theprocessing unit 1504 to perform any suitable embodiment of the methodsdescribed herein. For example, the processing unit 1504 can beconfigured to execute instructions causing one or more processors of theprocessing unit 1504 to analyze sensing data produced by the sensingmodule. The memory units can store sensing data from the sensing moduleto be processed by the processing unit 1504. In some embodiments, thememory units of the non-transitory computer readable medium 1506 can beused to store the processing results produced by the processing unit1504.

In some embodiments, the processing unit 1504 can be operatively coupledto a control module 1508 configured to control a state of the movableobject. For example, the control module 1508 can be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1508 can control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1504 can be operatively coupled to a communicationmodule 1510 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication can be used, such aswired communication or wireless communication. For example, thecommunication module 1510 can utilize one or more of local area networks(LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point(P2P) networks, telecommunication networks, cloud communication, and thelike. Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication module1510 can transmit and/or receive one or more of sensing data from thesensing module 1502, processing results produced by the processing unit1504, predetermined control data, user commands from a terminal orremote controller, and the like.

The components of the system 1500 can be arranged in any suitableconfiguration. For example, one or more of the components of the system1500 can be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 15 depicts asingle processing unit 1504 and a single non-transitory computerreadable medium 1506, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1500 can include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media can be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1500 can occur at one or more of theaforementioned locations.

As used herein A and/or B encompasses one or more of A or B, andcombinations thereof such as A and B.

While some embodiments of the present disclosure have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the disclosure. It should beunderstood that various alternatives to the embodiments of thedisclosure described herein may be employed in practicing thedisclosure. It is intended that the following claims define the scope ofthe invention and that methods and structures within the scope of theseclaims and their equivalents be covered thereby.

What is claimed is:
 1. A method of determining a flight path for anaerial vehicle, comprising: controlling the aerial vehicle to fly alonga first route; identifying, during the flight along the first route andwith aid of one or more processors, a change in a state of signaltransmission occurring at a first location, the change of the state ofsignal transmission indicating an abnormal state in a signaltransmission between the aerial vehicle and a control device; inresponse to identifying the change, determining, by the one or moreprocessors, a second location different from the first location;determining a second route to the second location; controlling, by theone or more processors, the aerial vehicle to fly to and land at thesecond location; and in response to determining that the signaltransmission between the aerial vehicle and the control device at thesecond location is changed from the abnormal state to a normal state,determining a third route, determining the third route including atleast one of: determining the third route to return to the first route;determining the third route to a starting point of the aerial vehicle;determining the third route based on a command from a user; determiningthe third route to include a location undetected by a sensor of theaerial vehicle during a previous flight; or determining the third routeto a service point providing one or more services to restore a normaloperational status.
 2. The method of claim 1, wherein the abnormal stateincludes a loss or weakening of a signal during the signal transmissionbetween the aerial vehicle and the control device.
 3. The method ofclaim 1, wherein determining the second route to the second locationincludes: determining the second location from one or more pointsdesignated by the user.
 4. The method of claim 1, further comprising:determining and marking in a map one or more usual landing points. 5.The method of claim 1, wherein the second location includes the servicepoint providing the one or more services to the aerial vehicle, the oneor more services including at least one of a part replacement service, acheck-up service, a tune-up service, or a repair service.
 6. The methodof claim 1, wherein in response to determining that the signaltransmission between the aerial vehicle and the control device at thesecond location is changed from the abnormal state to the normal state,determining the third route includes at least one of: determining thethird route to return to the first route; determining the third route tothe starting point of the aerial vehicle; or determining the third routebased on the command from the user.
 7. A method of determining a flightpath for an aerial vehicle, comprising: identifying, during flight andwith aid of one or more processors, a change in a state of signaltransmission occurring at a first location, the change of the state ofsignal transmission indicating an abnormal state in a signaltransmission between the aerial vehicle and a control device; inresponse to identifying the change, selecting a first destination withina proximity of a second location, wherein a state of signal transmissionat the second location during a previous flight is different from thestate of signal transmission at the first location; determining a firstflight path to reach the first destination; upon reaching the firstdestination by the aerial vehicle, assessing in real time the state ofsignal transmission at the first destination; and determining a secondflight path to a second destination based on the assessment including:in response to the state of the signal transmission at the firstdestination changing from the abnormal state to a normal state,determining the second flight path including at least one of:determining the second flight path to include a location undetected by asensor of the aerial vehicle during the previous flight; determining thesecond flight path to a predetermined location for the aerial vehicle toreach before identifying the change in the state of signal transmissionoccurring at the first location; determining the second flight path to astarting point of the flight; determining the second flight path to aservice point providing services to restore a normal operational status;or determining the second flight path to a location selected based on auser input at the control device or based on real-time information. 8.The method of claim 7, wherein the change in the state of signaltransmission includes a loss or weakening of a signal transmittedbetween the aerial vehicle and the control device.
 9. The method ofclaim 8, wherein determining the second flight path includes, inresponse to the signal transmission being lost or weakened at the firstdestination, including in the second flight path the location undetectedby the sensor of the aerial vehicle during the previous flight.
 10. Themethod of claim 8, wherein determining the second flight path includes,in response to the signal transmission being lost or weakened at thefirst destination, including in the second flight path a locationdetected by the sensor of the aerial vehicle during the previous flight.11. The method of claim 7, wherein in response to the state of thesignal transmission at the first destination changing from the abnormalstate to the normal state, determining the second flight path includesdetermining the second flight path to include location undetected by thesensor of the aerial vehicle during the previous flight.
 12. The methodof claim 7, wherein determining the first flight path includes, notincluding the location undetected by the sensor of the aerial vehicleduring the previous flight.
 13. The method of claim 7, wherein the firstflight path includes a reverse of a last flight path of the aerialvehicle.
 14. The method of claim 7, wherein: the first destination isselected based on at least one of the user input, an operational statusof the sensor onboard the aerial vehicle, or a current condition of aflight environment.
 15. The method of claim 7, wherein: the firstdestination is proximal to the second location with a preset distance;or the first destination is at the second location.
 16. The method ofclaim 7, wherein: the second location is a location detected by thesensor onboard the aerial vehicle during the previous flight.
 17. Themethod of claim 7, further comprising, prior to or concurrent withselecting the second destination: controlling the aerial vehicle tohover for a predetermined period of time to gather data in real time.18. The method of claim 17, wherein a starting point of the flight isselected as the second destination in response to no user input beingreceived within the predetermined period of time.
 19. The method ofclaim 1, wherein determining the third route further includes, inresponse to the signal transmission being lost or weakened at the secondlocation, including in the third route the location undetected by thesensor of the aerial vehicle during the previous flight.
 20. The methodof claim 1, wherein determining the third route further includes, inresponse to the signal transmission being lost or weakened at the secondlocation, including in the third route a location detected by the sensorof the aerial vehicle during the previous flight.